Crate jobslot

An implementation of the GNU make jobserver.

This crate is an implementation, in Rust, of the GNU make jobserver for CLI tools that are interoperating with make or otherwise require some form of parallelism limiting across process boundaries. This was originally written for usage in Cargo to both (a) work when cargo is invoked from make (using make’s jobserver) and (b) work when cargo invokes build scripts, exporting a jobserver implementation for make processes to transitively use.

The jobserver implementation can be found in detail online but basically boils down to a cross-process semaphore. On Unix this is implemented with the pipe syscall and read/write ends of a pipe and on Windows this is implemented literally with IPC semaphores.

Starting from GNU make version 4.4, named pipe becomes the default way in communication on Unix, which is supported by this crate.

However, Client::configure_and_run and Client::configure_make_and_run still use the old syntax to keep backwards compatibility with existing programs, e.g. make < 4.4.

To create a new fifo on unix, use Client::new_with_fifo and to use it for spawning process, use Client::configure_and_run_with_fifo or Client::configure_make_and_run_with_fifo.

The jobserver protocol in make also dictates when tokens are acquired to run child work, and clients using this crate should take care to implement such details to ensure correct interoperation with make itself.

Advantages over jobserver?

• jobslot contains bug fix for Client::configure is unsafe
• jobslot removed use of signal handling in the helper thread on unix
• jobslot uses winapi on windows instead of manually declaring bindings (some of the bindings seem to be wrong)
• jobslot uses getrandom on windows instead of making homebrew one using raw windows api
• jobslot::Client::from_env can be called any number of times on Windows and Unix.

Examples

Connect to a jobserver that was set up by make or a different process:

use jobslot::Client;

// See API documentation for why this is `unsafe`
let client = match unsafe { Client::from_env() } {
    Some(client) => client,
    None => panic!("client not configured"),
};

Acquire and release token from a jobserver:

use jobslot::Client;

let client = unsafe { Client::from_env().unwrap() };
let token = client.acquire().unwrap(); // blocks until it is available
drop(token); // releases the token when the work is done

Create a new jobserver and configure a child process to have access:

use std::process::Command;
use jobslot::Client;

let client = Client::new(4).expect("failed to create jobserver");
let mut cmd = Command::new("make");
let child = client.configure_and_run(&mut cmd, |cmd| cmd.spawn()).unwrap();

Features

• tokio: This would enable support of tokio::process::Command. You would be able to write:

  use tokio::process::Command;
  use jobslot::Client;
  
  let client = Client::new(4).expect("failed to create jobserver");
  let mut cmd = Command::new("make");
  let child = client.configure_and_run(&mut cmd, |cmd| cmd.spawn()).unwrap();

Caveats

This crate makes no attempt to release tokens back to a jobserver on abnormal exit of a process. If a process which acquires a token is killed with ctrl-c or some similar signal then tokens will not be released and the jobserver may be in a corrupt state.

Note that this is typically ok as ctrl-c means that an entire build process is being torn down, but it’s worth being aware of at least!

Windows caveats

There appear to be two implementations of make on Windows. On MSYS2 one typically comes as mingw32-make and the other as make itself. I’m not personally too familiar with what’s going on here, but for jobserver-related information the mingw32-make implementation uses Windows semaphores whereas the make program does not. The make program appears to use file descriptors and I’m not really sure how it works, so this crate is not compatible with make on Windows. It is, however, compatible with mingw32-make.

Structs

• Acquired
  An acquired token from a jobserver.
• AsyncAcquireClienttokio and Unix, or neither Unix nor Windows
  Extension of [Client] that supports async acquire.
• Client
  A client of a jobserver
• TryAcquireClient
  Extension of Client that supports non-blocking acquire.

Enums

• IntoTryAcquireClientError
  Possible errors for Client::into_try_acquire_client

Traits

• Command
  Command that can be accepted by this crate.